Many image capture devices use solid-state image sensors containing a two-dimensional array of pixels. Increasingly, these image sensors are active pixel sensors (APS), also known as CMOS sensors, because they can be fabricated at high resolution using a Complementary Metal Oxide Semiconductor (CMOS) process.
In cameras and other types of image capture devices, CMOS image sensors often control exposure of each pixel by using an electronic rolling shutter, illustrated in the left hand portion of FIG. 1. In this figure, a two-dimensional pixel array 610 is read out a row of pixels at a time, indicated by the arrow labeled scan direction. This read out proceeds row by row with regular timing between the readout of one row and the readout of the next. For example, if a pixel array has 2000 rows and is read out in 30 milliseconds, the row time would be 15 microseconds. This time from the readout of one row to the readout of the next row is called the row time.
The exposure or integration time for each row of pixels is controlled electronically by resetting the row of pixels for a specific amount of time before the row is read out. The exposure time is denoted by texp, and is usually an integer multiple of the row time. Thus, FIG. 1 shows a row reset pointer 612 that is 10 rows ahead of a row readout pointer 614. In this case, texp is 10 times the row time.
A consequence of rolling shutter exposure control is that while each row of the image can have an identical amount of exposure time, each row is exposed at a slightly different time. The rows at the top of the imager are exposed and read out before the rows at the bottom of the imager are exposed and read. The exposure time window that is alloted for each row of the imager moves later in time by the row time.
To the left of the graph of FIG. 1, time is shown in units of rows. The row time can be used to convert the time axis from row units to other units, such as milliseconds. For example, FIG. 1 illustrates an imager operating with a row time of 15.6 microseconds, so the time required to read 1600 rows is rounded off to about 25 milliseconds.
Flickering illuminants, such as fluorescent lights, provide an illumination that changes rapidly and periodically over time with cyclic power fluctuation. This flicker is illustrated in the right hand side graph of FIG. 1, showing relative illumination changing over time from about zero to full power (with relative illumination changing between 0.0 and 1.0) in a curve 616. In this example the illuminant power fluctuates with a period of about 8.33 milliseconds, or 1/120 second, such as would happen with a fluorescent light powered with line current having a 1/60 second period. Those skilled in the art will appreciate that an actual fluorescent light does not provide an illumination power that is precisely a rectified sinusoid like curve 16; however, this idealized example illustrates the illuminant flicker problem.
The interaction of the illuminant flicker and the rolling shutter readout timing results in bands of light and dark exposure in the image, depending on the value of exposure time texp compared with the period of the illuminant flicker. For example, if exposure time texp is equal to 1 row time, then the dark and light bands follow the illuminant curve very closely. Conversely, it is well known in the art that setting exposure time texp to be one full period of the illuminant flicker eliminates the exposure bands completely, since each row of the image then integrates light from one full cycle of the illumination flicker.
FIG. 2A illustrates the magnitude of the exposure banding arising from illuminant flicker with different exposure times. In FIG. 2A, solid dark curve 616, the same curve as in FIG. 1, shows the relative illuminant power for an idealized fluorescent light. A rectangle 618 labeled Short illustrates the length of the exposure time used to produce the dotted curve 620 that cycles between about 0.4 and 0.9 times relative illumination. Because this short exposure time includes less than a full cycle of illuminant flicker, there is some amount of banding perceptible in the exposure at the image sensor. A rectangle 622 labeled Full Cycle in FIG. 2A illustrates the length of the exposure time used to produce a dashed curve 624 that is constant at approximately 0.64. This exposure exhibits no banding, because each row of the image sensor is exposed for one full cycle of the illuminant flicker. The rectangle 626 labeled Long illustrates a longer exposure time, used to generate the dashed-dotted curve 628 that cycles between 0.57 and 0.72. This longer exposure again has exposure bands, because each line is exposed for a time that is not an integer multiple of the flicker illumination period.
The graph of FIG. 2B shows relative banding as related to exposure time for 60 Hz fluctuation in light intensity. At very low exposure times, banding effects are most pronounced. At a small number of exposure times, depending on the frequency rate of the illuminant flicker, there is no banding or perceptible banding is very low. As exposure time increases beyond about 50 msec, banding is also reduced. The graph of FIG. 2C shows this relationship for 50 Hz fluctuation.
It is well known in the art that selecting an exposure time that is an integer multiple of the illumination flicker period helps to avoid exposure bands, as illustrated by curve 624 in FIG. 2A. For example, U.S. Pat. No. 7,142,234 B2 and U.S. Patent Application Publication No. 2005/0046704 A1 both teach how to avoid flicker when using rolling shutter readout, by selecting an exposure time than is an integer multiple of the period of illuminant flicker. Given that illuminant flicker is nearly always controlled by the frequency of alternating current (AC) electrical power in a geographic region, this usually simplifies to one of three cases: no illuminant flicker, 100 Hz flicker, or 120 Hz flicker. With an exposure control system that can eliminate the effects of illuminant flicker by controlling exposure time, the primary remaining problem is simply detecting the illuminant flicker and its frequency.
There are a number of conventional approaches for determining whether there is illuminant flicker and what frequency it has. For example, U.S. Pat. No. 7,142,234 to Kaplinsky et al., U.S. Pat. No. 7,187,405 to Poplin et al., and U.S. Pat. No. 6,710,818 to Kasahara et al. all teach methods for detecting illuminant flicker in a system that uses a rolling shutter. U.S. Pat. No. 7,142,234 uses a frequency analysis approach, computing a frame difference and checking for a peak signal at the expected frequency. Alternately, U.S. Pat. No. 7,187,405 discloses the approach of computing a difference between frames, then performing a frequency analysis or a cross-correlation between a difference vector and a sinusoid of the expected frequency. U.S. Pat. No. 6,710,818, meanwhile, discloses a technique of taking column vectors from adjacent frames, dividing one by the other, then performing a Discrete Fourier Transform (DFT) analysis on the ratio that is obtained. U.S. Pat. No. 6,710,818 also discloses various filtering techniques for smoothing the ratio vector to improve detection reliability.
Conventional approaches such as those just cited have several limitations. In practice, frame differences provide a poor indication of variation in exposure, particularly when these values are applied to linear image sensor data and the task of acquiring frame difference data can be computationally costly. Ratios of linear data provide a more sensitive result, but computational division is also relatively expensive in terms of the needed processing resources and time.
Frequency analysis methods are relatively computationally expensive in that a vector is projected onto one or more basis functions, requiring storage and processing capacity to provide these basis functions. Further, frequency analysis works effectively with exposure bands that project cleanly onto one or more basis functions. When a non-integer number of flicker cycles are included in a frame, the discontinuity from one frame to the next can corrupt the frequency response results.
Earlier methods for detecting and compensating for illuminant banding all have limitations, and there is a need for flicker detection with improved reliability and efficiency. There is, then, a need for a technique that is sensitive to exposure variation and is less demanding of computing resources.